The Challenge of Evaluating Electronic Decision Support in the Community

There is clear room for improvement in all existing clinical decision support systems (CDSS) to more closely meet the needs of their target users and work processes. Ongoing evaluation serves not just to document benefits, but also to provide feedback to the CDSS engineering process to create more usable and effective tools. This paper discusses the challenges of achieving evaluation data that provides detail on system use at the point of care with a focus on improvement of decision support in a community setting such as a General Practice clinic. After describing some results of CDSS evaluation from the literature, as well as our own recent efforts at laboratory evaluation of the PREDICT CVD/Diabetes tool, we identify specific challenges that arise in CDSS evaluation. We suggest and discuss several approaches that may lead to facilitated and improved data gathering to help CDSS developers understand how their tools are being used and hence how to improve them, including: (a) the challenges and opportunities afforded by automated logging of CDSS usage; and (b) the value of an electronic health record interchange standard for the creation of realistic test cases.